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FIELD OF THE INVENTION 

[0001] The invention relates to digital image sensor systems 
and, in particular, to a method for subtracting fixed pattern 
noise in a CMOS digital image sensor. 

DESCRIPTION OF THE RELATED ART 

[0002] A digital imaging system for still or motion images 
uses an image sensor or a photosensitive device that is sensitive 
to a broad spectrum of light to capture an image of a scene. The 
photosensitive device reacts to light reflected from the scene 
and can translate the strength of that light into electronic 
signals that are digitized. Generally, an image sensor includes 
a two-dimensional array of light detecting elements, also called 
pixels, and generates electronic signals, also called pixel data, 
at each light detecting element that are indicative of the 
intensity of the light impinging upon each light detecting 
element. Thus, the sensor data generated by an image sensor is 
often represented as a two-dimensional array of pixel data. 

[0003] A CMOS image sensor with pixel level analog-to-digital 
conversion is described in U.S. Patent No. 5,461,425 of B. Fowler 
et al. (the x 425 patent). Such an image sensor, referred to as a 
digital pixel sensor (DPS) , provides a digital output signal at 
each pixel element representing the light intensity detected by 
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that pixel element. The combination of a photodetector and an 
analog-to-digital (A/D) converter in an area image sensor helps 
enhance detection accuracy, reduce power consumption, and 
improves overall system performance. 

[0004] CMOS image sensors generally exhibit fixed pattern 
noise that is visible as a stationary image pattern superimposed 
on captured images. Fixed pattern noise is considered one of the 
major weaknesses and deficiencies of CMOS digital image sensors. 
Fixed pattern noise is the result of multiple effects that yield 
variations and nonunif ormity in individual pixel performance and 
response to light. 

[0005] First, CMOS image sensors typically suffer from 
temperature dependent dark current that reduces the signal to 
noise ratio of the image sensor and limits the sensitivity of the 
image sensor. Dark current generally relates to the leakage 
current from the photodiodes or photo- transistors that are used 
as the light detecting elements in the image sensor. To make 
matters worse, the dark current is not spatially uniform but 
rather is randomly distributed and varies as a function of pixel 
location on the image sensor. As a result, when a CMOS image 
sensor is used to capture an image, the dark current associated 
with each pixel becomes a fixed pattern noise image superimposed 
on the desired image. The fixed pattern noise image is 
particularly visible in dark scenes and results in reduced 
sensitivity of the image sensor. 

[0006] Other factors contributing to fixed pattern noise in a 
CMOS image sensor include spatially non-uniform capture of light, 
spatial variation in analog-to-digital conversion, and spatial 
parametric variation in the pixel circuitry in the image sensor. 



These factors can be a result of spatial variations in the 
optical filter structure on top of the sensor, spatial variation 
in the micro lenses on top of the sensor, spatial variation in 
the conversion efficiency of the photodiodes, voltage drop across 
the pixel array, and spatial variation in the transistor 
threshold voltage, transistor gain and geometries. 

[0007] Fixed pattern noise in a CMOS image sensor is a unique 
image for each image sensor and the fixed pattern noise may vary 
as a function of image brightness, temperature, operating 
voltage, and other operating parameters. To increase the 
sensitivity of the image sensor and to improve the quality of the 
captured images, it is desirable to remove the fixed pattern 
noise from the captured images to increase the sensitivity of the 
image sensor and the quality of the captured images. In general, 
fixed pattern noise is time invariant and can be removed from a 
captured image if the pattern is known. However, because the 
fixed pattern noise can vary with various operating conditions, 
the fixed pattern noise image will vary accordingly making 
elimination of the image difficult. 

[0008] Conventional methods for fixed pattern noise 
subtraction are unsatisfactory. U.S. Patent 6,061,092 discloses 
a method for eliminating fixed pattern noise by storing a large 
number of fixed pattern noise images, also called dark images, in 
a host computer and then retrieving the dark images from the host 
computer based on different operation parameters of the camera. 
Such method requires large amount of memory and requires the 
image sensor to be tethered to a host computer which is not 
practical for portable applications. U.S. Patent 6,535,617 
discloses a method for removal of fixed pattern noise by 



- 3 - 



estimating the fixed noise pattern from a composite image which 
composite image is derived by summing multiple image frames 
captured at a uniform illumination. Then, for each frame of 
image, the fixed pattern noise is estimated as a dot product of 
the image frame and the normalized filtered composite image. 
This method is not satisfactory because it does not take into 
consideration changes in the fixed pattern noise due to operating 
parameters of the image sensor. For example, fixed pattern noise 
usually varies with temperature. As temperature changes, the 
fixed pattern noise either becomes more pronounced or fades away. 
A fixed pattern noise removal method that does not take into 
consideration the temperature variation of the fixed pattern 
noise is ineffective in removing the noise component in a 
captured image. 

[0009] Therefore, an improved method for accurately 
determining fixed pattern noise and subtracting the fixed pattern 
noise from images captured by a digital image sensor is desired. 

SUMMARY OF THE INVENTION 

[0010] According to one embodiment of the present invention, a 
method for subtracting fixed pattern noise in a digital imaging 
system incorporating a digital image sensor includes: acquiring a 
reference image of the digital image sensor when the digital 
image sensor receives no illumination, storing a reference value 
of an operating parameter associated with the reference image 
where the reference image is indicative of the fixed pattern 
noise associated with the digital image sensor when the operating 
parameter has the reference value, storing a model describing the 
behavior of the fixed pattern noise as a function of the 
operating parameter. Then, the method continues with acquiring a 
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first image, measuring a current value of the operating parameter 
associated with the first image, calculating a noise prediction 
image by extrapolation of the reference image in accordance with 
the model and based on the current value and the reference value 
of the operating parameter, and subtracting the noise prediction 
image from the first image to generate a final image. 

[0011] In one embodiment, the fixed pattern noise subtraction 
method of the present invention is implemented in a video imaging 
system including a digital image sensor and a digital image 
processor. 

[0012] The present invention is better understood upon 
consideration of the detailed description below and the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] Figure 1 is a flow chart illustrating the fixed pattern 
noise subtraction method according to one embodiment of the 
present invention . 

[0014] Figure 2 is a flow chart illustrating the method for 
acquiring a reference image of the fixed pattern noise according 
to one embodiment of the present invention. 

[0015] Figure 3 is a block diagram of a video imaging system 
based on Figure 1 of U.S. Patent Application Serial No. 
10/634,3 02 and modified to implement the fixed pattern noise 
subtraction method according to one embodiment of the present 
invention. 
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[0016] Figure 4 is a block diagram of a digital image sensor 
as described in U.S. Patent No. 5,461,425 of Fowler et al . 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0017] In accordance with the principles of the present 
invention, a method for fixed pattern noise subtraction is 
implemented in a digital imaging system incorporating a digital 
image sensor to estimate and remove fixed pattern noise from a 
captured image. The fixed pattern noise subtraction method 
involves acquiring a reference image indicative of the fixed 
pattern noise of the image sensor, storing reference values of 
operating parameters associated with the reference image, 
capturing an image, measuring the current values of operating 
parameters of the captured image, and calculating a prediction of 
the fixed pattern noise for the captured image by extrapolating 
from the reference image where the extrapolation is based on a 
model of the fixed pattern noise as a function of the values of 
the operating parameters. The noise prediction can then be 
subtracted from the captured image to generate a final image free 
of fixed pattern noise. In one embodiment, the subtraction is 
performed only when the captured image is a low illumination 
image, such as an image with pixel data at or below a given 
illumination threshold. The fixed pattern noise subtraction 
method of the present invention is capable of almost completely 
removing all fixed pattern noise associated with pixels in a 
digital image sensor. When the fixed pattern noise subtraction 
method of the present invention is implemented in an image 
sensor, the sensitivity of the image sensor can be greatly 
enhanced, especially for low light images. 
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[0018] Figure 1 is a flow chart illustrating the fixed pattern 
noise subtraction method according to one embodiment of the 
present invention. Fixed pattern noise subtraction method 10 can 
be implemented in any digital imaging systems incorporating a 
digital image sensor and is particularly useful when implemented 
in imaging systems using CMOS image sensors for eliminating fixed 
pattern noise. Generally, the fixed pattern noise subtraction 
method of the present invention is an open loop algorithm. Fixed 
pattern noise subtraction method 10 of the present embodiment 
performs calibration of the image sensor either once during in- 
system manufacturing, once during image sensor manufacturing, or 
once every time the image sensor is powered up to obtain a 
reference image and that reference image is used for fixed 
pattern noise removal in all images subsequently captured by the 
image sensor. When the calibration is performed once during 
image sensor manufacturing, some kind of tracking between a 
specific image sensor and the associated reference image is 
required. For example, a "birth certificate" for each image 
sensor can be used to associate an image sensor with the 
reference image. 

[0019] Referring to Figure 1, fixed pattern noise subtraction 
method 10 starts by calibrating the image sensor. Calibration of 
the image sensor involves acquiring a reference image of the 
fixed pattern noise (step 12) and storing the reference values of 
operating parameters associated with the reference image when the 
image was taken (step 14) . In the present embodiment, 
calibration is only performed once for each image sensor and is 
usually performed in- system for each image sensor. That is, the 
calibration is performed after the image sensor is incorporated 
in an imaging system, such as a video camera or a still-motion 
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digital camera. The reference image indicative of the fixed 
pattern noise associated with that image sensor is stored in the 
imaging system so that the reference image can be used 
subsequently to eliminate fixed pattern noise from captured 
images for that image sensor. When the calibration is only 
performed once, the reference image should be stored in a 
permanent or non-volatile memory of the imaging system, such as a 
flash memory. It is instructive to note that the fixed pattern 
noise subtraction method of the present invention requires only 
the storage of one reference image and associated reference 
values. Therefore, it is possible to store the data in the 
imaging system in which the image sensor is incorporated. The 
fixed pattern noise subtraction method of the present invention 
does not require the use of a large amount of memory or a host 
computer to store the reference image. 

[0020] In another embodiment, calibration is performed for the 
image sensor every time the image sensor or the imaging system in 
which the image sensor is incorporated is powered up. 
Alternately, the user of the imaging system can initiate the 
calibration. In these cases, a temporary memory or volatile 
memory, such as a SRAM, can be used to store the reference image 
and the associated reference values. In yet another embodiment, 
calibration is performed at image sensor manufacturing time. A 
"birth certificate" associated with each image sensor is used to 
associate an image sensor with the reference image data belonging 
to that image sensor. 

[0021] In one embodiment, the calibration process of the image 
sensor is performed as follows. Figure 2 is a flow chart 
illustrating the method for acquiring a reference image of the 
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fixed pattern noise according to one embodiment of the present 
invention. Referring to Figure 2, a series of dark images are 
captured using the image sensor (step 40) . The dark images can 
be taken by closing the aperture of the imaging system so that 
the image sensor is not exposed to any light. The reference 
image is generated by averaging the pixel values of the multiple 
dark images for each pixel (step 42) . By averaging multiple dark 
images, random noise components of the dark images can be 
removed. The resulting reference image is indicative of the 
fixed pattern noise associated with the image sensor and can be 
stored for use in fixed pattern noise removal later (step 44) . 

[0022] Returning to Figure 1, the calibration process of fixed 
pattern noise subtraction method 10 of the present invention 
continues with storing the reference values of operating 
parameters associated with the reference image being captured 
(step 14) . Because the fixed pattern noise of the image sensor 
can vary with operating conditions such as temperature, 
illumination level, and exposure time, these variations need to 
be accounted for so that accurate fixed pattern noise removal can 
be performed. In accordance with the present invention, the 
reference values of the operating parameters are stored so that 
the parameters can be used to predict accurate noise values of 
the captured images. 

[0023] In the present embodiment, the operation parameter 
being measured and stored include the dark current of the 
photodetectors which dark current is used as an indicator for the 
operating temperature of the image sensor. Dark current is the 
leakage current of "dark pixels" in the image sensor. Dark 
pixels are permanently and totally shadowed, such as covered by a 



- 9 - 



metal layer. The image sensor may include one dark pixel or one 
or more columns or rows of dark pixels. Generally, a more 
accurate dark current measurement can be made by measuring the 
dark current of multiple dark pixels and averaging the dark 
current values to remove random noise. The leakage current of a 
photodiode or a phototransistor has a known dependency on 
temperature. Therefore, it is possible to use the amount of 
leakage current of the dark pixels when the reference image is 
captured as an indicator of the temperature at which the 
reference image is captured. Note that the dark current 
measurement is also dependent on other operating conditions, such 
as voltages. Some of these effects may also, at least partially, 
cancel out. However, in most applications, the dark current 
measurement is a sufficiently good indicator of the temperature 
of the image sensor. 

[0024] The calibration of the image sensor is completed when 
the reference image for the image sensor is acquired and the 
reference values of the operating parameters associated with the 
reference image is stored. To implement fixed pattern noise 
subtraction method 10, the imaging system in which the image 
sensor is incorporated also needs to store a system model 
describing the behavior of the fixed pattern noise as a function 
of the operating parameters (step 16) . The system model can be 
derived from known characteristics of photodiodes and 
phototransistors . The system model can also be derived 
experimentally by analyzing data collected from a large number of 
image sensors. Based on the data acquired for a large number of 
images from multiple image sensors taken at various temperature, 
operating voltage and exposure times, the fixed pattern noise can 
then be modeled as a function of the operating parameters. The 
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process of deriving a system model for the fixed pattern noise is 
not critical to the practice of the present invention. Fixed 
pattern noise subtraction method 10 only requires a model 
describing the behavior of the fixed pattern noise as a function 
of the operating parameters to be stored in the imaging system. 

[0025] In one embodiment, the system model used in the fixed 
pattern noise subtraction method 10 describes the behavior of the 
fixed pattern noise as a function of one or more of the following 
operating parameters: (1) the nature of exposure, such as the 
amount of light impinging on the image sensor and the wavelength 
of the incident light; (2) the duration of the light exposure; 

(3) the temperature of the image sensor as indicated by the 
leakage current or dark current; (4) location of the pixels in 
the image sensor; (5) the reset voltage of the photodetectors; 
and (6) the operating voltage of the image sensor. 

[0026] In the present embodiment, fixed pattern noise 
subtraction method 10 of the present invention utilizes only the 
temperature, as measured by the average dark current of the dark 
pixels, as the operating parameter of interest. Because the dark 
current has a well-known exponential dependency on temperature, 
the dark current can be used as an effective indicator of 
temperature. Thus, the imaging system stores a system model of 
the fixed pattern noise variation with respect to temperature. 

[0027] After calibration of the image sensor is performed, the 
image sensor and the imaging system in which the image sensor is 
incorporated can be used to capture images. At step 18, the 
image sensor is activated to capture an image of interest. Fixed 
pattern noise subtraction method 10 proceeds to measure the 
current values of the operating parameters (step 20) , that is, 
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the values of the operating parameters associated with the 
current image being captured. In the present embodiment, the 
operating parameter of interest is temperature and fixed pattern 
noise subtraction method 10 ascertains the current operating 
temperature by measuring the leakage current of the dark pixels 
in the image sensor. 

[0028] Then, a noise prediction for each pixel of the current 
image is calculated (step 22) . The noise prediction is 
calculated by extrapolating the pixel values of the reference 
image based on the current values of the operating parameters as 
compared to the reference values of the operating parameters. 
Specifically, the extrapolation applies the stored system model 
of the fixed pattern noise as a function of the operating 
parameters. In the present embodiment, the extrapolation is 
performed based on a comparison of the temperature of the 
reference image and the temperature of the captured image, as 
measured by the dark current of the image sensor. The stored 
system model of the fixed pattern noise as a function of 
temperature is used to determine what the pixel values for the 
reference image should be when the temperature varies from the 
reference temperature level to the current temperature level . 

[0029] Using the system model of the fixed pattern noise as a 
function of temperature, a noise prediction for the current image 
of the image sensor is calculated. The noise prediction can be 
subtracted from the captured image to generate a final image free 
of fixed pattern noise. 

[0030] In the present embodiment, fixed pattern noise 
subtraction is only activated when the illumination is below a 
certain threshold level since fixed pattern noise is visible 
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mainly in low light conditions. Referring to Figure 1, fixed 
pattern noise subtraction method 10 determines the illumination 
level of the captured image to determine if it is above or below 
a pre-selected threshold (step 24) . The illumination level of 
the captured image can be measured by taking an average of pixel 
values of selected pixels which pixels are distributed through 
the image. When the illumination level of the captured image is 
above the threshold, no fixed pattern noise subtraction is 
performed and fixed pattern noise subtraction method 10 returns 
to step 18 to await the capture of another image. 

[0031] If the illumination level of the captured image is at 
or below the threshold, then fixed pattern noise subtraction 
method 10 proceeds to step 26 where the noise prediction 
calculated previously for each pixel is subtracted from the 
respective pixel values of the captured image to provide a final 
image free of fixed pattern noise. Then, fixed pattern noise 
subtraction method 10 returns to step 18 to await the capture of 
another image. 

[0032] In yet another embodiment, a gradual activation of the 
fixed pattern noise subtraction is applied. Specifically, 
depending on the illumination or temperature condition, a 
variable number of bits may be allocated for storing the noise 
prediction data relating to the fixed pattern noise. When the 
illumination is at or below a certain threshold (that is, the 
scene gets darker) , the pixel data for the real image becomes 
less important and can be expressed in fewer bits while the fixed 
pattern noise becomes more visible and can be expressed in more 
bits. Thus, depending on the illumination conditions, variable 
number of bits can be allocated for the noise prediction data so 
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that more precise noise prediction values can be applied when the 
fixed pattern noise is more visible. 

[0033] For instance, a full m-bit of the noise prediction 
value is applied when the temperature is above a predetermined 
temperature level while fewer than m bits of noise prediction 
value is applied when the temperature is below the predetermined 
temperature level. Alternately, the fixed pattern noise 
subtraction method of the present invention can apply a full 
m-bit of the noise prediction value only when the light intensity 
is below a certain level where the level is below the 
pre-selected threshold used in step 24. When the light intensity 
is above the threshold level, fewer than m bits of noise 
prediction is used and the unused bits can be used to store pixel 
data relating to the real image. Specifically, the application 
of the noise prediction can be tailored so that full m-bit noise 
subtraction is performed only in conditions where fixed pattern 
noise is known to be most prevalent. In this manner, the pixel 
values of the captured image is preserved as much as possible and 
noise subtraction is performed only when fixed pattern noise 
substantially contribute to the degradation of the captured 
images . 

[0034] The fixed pattern noise subtraction method of the 
present invention can be applied to a variety of digital imaging 
systems incorporating digital image sensors. In one embodiment, 
the fixed pattern noise subtraction method of the present 
invention is applied in a video imaging system including a 
digital image sensor component and a digital image processor 
component. Such a video imaging system is described in copending 
and commonly assigned U.S. patent application serial no. 
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10/634,3 02, entitled "A Video Imaging System Including A Digital 
Image Sensor And A Digital Signal Processor," filed August 4, 
2003, of Michael Frank et al . , which patent application is 
incorporated herein by reference in its entirety. 

[0035] Figure 3 is a block diagram of a video imaging system 
based on Figure 1 of the aforementioned '3 02 patent application 
and modified to implement the fixed pattern noise subtraction 
method according to one embodiment of the present invention. In 
brief, digital image sensor 102 of video imaging system 100 is an 
operationally "stand-alone" imaging system operative to capture 
image data and provide entire frame of image data to digital 
image processor 104 for processing. Digital image processor 104 
includes a frame buffer for storing the image data transferred 
from digital image sensor 102 and processing the image data based 
on instructions from system processor 240. 

[003 6] In one embodiment, image sensor 102 is a digital image 
sensor implemented as a digital pixel sensor (DPS) , that is a 
CMOS image sensor with pixel level analog-to-digital conversion 
capabilities, as described in the aforementioned x 425 patent. In 
the present description, a digital pixel sensor (DPS) array or a 
sensor array refers to a digital image sensor having an array of 
photodetectors where each photodetector produces a digital output 
signal. In one embodiment of the present invention, the DPS 
array implements the digital pixel sensor architecture 
illustrated in Figure 4 and described in the aforementioned '425 
patent. The DPS array of the M25 patent utilizes pixel level 
analog-to-digital conversion to provide a digital output signal 
at each pixel. The pixels of a DPS array are sometimes referred 
to as a "sensor pixel" or a "sensor element" or a "digital 
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pixel/' which terms are used to indicate that each of the 
photodetectors of a DPS array includes an analog-to-digital 
conversion (ADC) circuit, and is distinguishable from a 
conventional photodetector which includes a photodetector and 
produces an analog signal. The digital output signals of a DPS 
array have advantages over the conventional analog signals in 
that the digital signals can be read out at a much higher speed. 
Of course, other schemes for implementing a pixel level A/D 
conversion in an area image sensor may also be used in the image 
sensor of the present invention. 

[0037] When fixed pattern noise subtraction method 10 of the 
present invention is incorporated in video imaging system 100 of 
Figure 3, the reference image and reference values can be stored 
in a memory buffer 242 accessible by system processor 24 0 for 
generating the noise prediction for each pixel value. In one 
embodiment, the noise prediction is stored in a memory buffer 
separate from frame buffer 228 of digital image processor 104. 
For example, the noise prediction can be stored in memory buffer 
242 as well. However, in an alternate embodiment of the present 
invention, a unified memory, such as the frame buffer, is used to 
store the pixel values of the noise prediction as well as the 
pixel values of the captured image. Using a unified memory to 
store the noise prediction as well as the captured image has the 
advantage of efficient memory usage by allowing the reuse of 
memory locations in the frame buffer not used by the captured 
image to store the reference image. For instance, when digital 
image processor 104 stores pixel data in m bits, a captured image 
taken at low light conditions can be expressed in less than m 
bits. Thus, when the captured image and the noise prediction are 
stored in the same memory, such as frame buffer 22 8, memory 
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locations not used to store the captured image can be used to 
store more bits of the noise prediction so that more precise 
noise prediction values can be stored, 

[0038] In one embodiment, frame buffer 228 is organized to 

store 12 bits per pixel. The frame buffer is partitioned into 
two bit fields: one bit field is used to store the captured 
image, also referred to as the image component, and the other bit 
field is used to store the noise prediction, also referred to as 
the noise component. A programmable number of bits can be 
assigned to each bit field. In operation, the image component of 
the frame buffer is updated with image data from digital image 
sensor 102. The noise component of the frame buffer is 
write-protected so that they will not be overwritten. Instead, 
system processor 24 0 updates the noise component with noise 
prediction values generated using the reference image stored in 
memory buffer 242 and the current and reference values of the 
operating parameters and by applying the system model of the 
noise relationship to the operating parameters. 

[0039] In digital image processor 104, the subtraction of the 
noise component from the image component can be carried out using 
an arithmetic unit. Alternately, a lookup table can be used to 
perform the subtraction. In one embodiment, lookup table 23 0 is 
used to perform the subtraction of the noise component from the 
image component. Specifically, lookup table 230 subtracts n bits 
of noise component from 12 -n bits of image component to generate 
the final pixel values and converts the final pixel values into 
binary 14 bit pixel values. The advantage of using a lookup 
table to perform the subtraction is that other complex operation 
can be performed at the same time as the subtraction. 
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[0040] When the fixed pattern noise subtraction method of the 
present invention is implemented in video imaging system 100, 
effective and accurate fixed pattern noise removal can be 
realized. The sensitivity of video imaging system 100 can be 
greatly improved, particularly for low illumination images. 

[0041] The above detailed descriptions are provided to 
illustrate specific embodiments of the present invention and are 
not intended to be limiting. Numerous modifications and 
variations within the scope of the present invention are 
possible. For example, in the above description, the fixed 
pattern noise subtraction method of the present invention is 
implemented in a video imaging system. This is illustrative only 
and in other embodiments, the fixed pattern noise subtraction 
method of the present invention can be implemented in any digital 
imaging systems for still or motion images. Furthermore, the 
video imaging system described above can be modified for use as a 
still motion camera. Fixed pattern noise subtraction method of 
the present invention can be applied effectively in either still 
or motion imaging systems. The present invention is defined by 
the appended claims. 
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